What is the right course of action here?

You chose the correct option by allowing the yellow vehicle to pass first. The traffic sign on your side is a "yield" sign, which means you must give way to other vehicles on the main road. The yellow vehicle is already on the main road and going straight, so it has the right of way. Your vehicle must wait until the yellow vehicle has passed before you can complete your turn. The red car is waiting and is not yet in your immediate path, so you do not need to concern yourself with it first.

Remember, when you see a yield sign, you must always give priority to vehicles on the intersecting road unless there is a traffic light or other instructions.